If you sat through an MES vendor briefing in the last year, you saw a copilot demo. Siemens has been building generative AI into Opcenter and its broader Industrial Copilot lineup. SAP has been pushing Joule into S/4HANA manufacturing workflows and adjacent MES touchpoints. Rockwell Automation has talked up AI assistance across FactoryTalk. AVEVA has done the same across its MES and operations portfolio. The pitch is consistent everywhere: ask a question in plain English, get an answer pulled from your production data, skip the report-building.
What’s much less consistent is what’s actually running under the hood. Some of these features are genuinely wired into the MES data model — pulling from work order records, genealogy, routing, and quality holds through governed queries. Others are a chat window bolted onto a general-purpose language model with a thin retrieval layer over whatever documents someone indexed. Both will look similar in a scripted demo. They behave very differently at 2 a.m. when a line goes down and someone actually needs an answer.
That gap matters more this year than it has before, because renewal cycles are landing right as vendors push these features to general availability. If you’re evaluating a copilot addendum on your contract, or comparing it against a competitor’s roadmap slide, you need a way to test it that doesn’t depend on the vendor’s own script.
Why the demo doesn’t tell you what you need to know
Vendor demos are, reasonably, built around clean data and best-case questions. They’re rehearsed against a golden dataset, run on a network with no latency issues, and phrased in the exact way the model was tuned to expect. None of that resembles a real plant, where work-order data is inconsistent across lines, where operators phrase questions in shorthand specific to their shift, and where the answer that matters is the one that comes with a caveat attached — “here’s the likely cause, but confirm against the calibration log” — rather than a confident, wrong one.
The honest way to evaluate a copilot is to stop watching the vendor drive and put your hands on the keyboard, ideally in a sandbox or pilot environment connected to something close to your real MES instance. Here are five tasks that expose the differences that actually matter on a shop floor.
1. Root-cause query
Ask it something a controls engineer would actually ask: why did a specific work order run over cycle time, or what’s common across the last several scrap events on a given line. A copilot with real MES integration should be able to cross-reference genealogy, downtime codes, and quality data and give you a scoped, sourced answer — ideally showing which records it pulled from. A bolt-on chatbot will often give you a plausible-sounding narrative that isn’t actually traceable to your data at all. Ask it to show its work. If it can’t point to the specific records behind its answer, treat the output as a guess, not a finding.
2. Work-order drafting
Have it draft a new work order from a natural-language request — say, a short run of a variant product with a modified routing step. This tests whether the copilot understands your ISA-95 data model well enough to populate real fields correctly, or whether it’s just generating plausible-looking text that a human then has to rebuild by hand anyway. Check whether it respects your existing approval workflows rather than trying to push a draft straight to the floor.
3. Spec lookup
Ask for a specific tolerance or material spec buried in a document that’s actually messy — a scanned PDF, a revision with tracked changes, an older spec superseded by a newer one. This is where retrieval quality shows up fast. A well-built copilot will cite the document and revision. A weaker one will confidently quote an outdated spec because it couldn’t distinguish document versions in its index.
4. Changeover instructions
Ask it to walk through a changeover sequence for a specific SKU pairing. Good implementations tie this back to your actual standard work instructions and equipment parameters, ideally referencing an ISA-88 recipe or equivalent, not generic best-practice text. If the instructions it gives don’t match your validated procedures exactly, that’s not a minor gap — it’s a safety and quality risk waiting to happen.
5. Exception escalation
Trigger a quality hold or downtime scenario and see whether the copilot can correctly identify who should be notified, through what channel, and with what information, based on your actual escalation matrix. This is the task that separates a real operations copilot from a Q&A bot — escalation logic requires the tool to understand roles, shifts, and authority levels, not just retrieve text.
Scoring what you see
For each task, score three things separately: accuracy against your actual data, traceability (can it show its sources), and whether it respects your existing governance and workflow rather than trying to route around it. A copilot that scores well on traceability but slower on raw fluency is, in our assessment, the safer bet for a regulated or high-mix environment. A copilot that sounds impressively conversational but can’t cite its sources is the one to be wary of, no matter how good the demo looked.
It’s also worth testing failure gracefully. Ask something the system genuinely shouldn’t be able to answer — a question about a plant it has no data access to, or a request that would violate a segregation-of-duties rule. A trustworthy copilot says it doesn’t know or that the action requires approval. One that fabricates an answer to stay helpful is a bigger liability than having no copilot at all.
Where this leaves buyers in 2026
The major MES vendors are converging on the same broad approach — natural-language access to production data, grounded in the platform’s own model rather than a generic assistant — but they’re at different points of maturity in how deeply that access reaches into governed workflows versus how much is still document retrieval dressed up as conversation. That’s a fair place for the market to be; this is a genuinely new category, and depth of integration takes time to build regardless of vendor size or reputation.
What it means for you is that the copilot line item on a renewal shouldn’t be evaluated on the strength of a roadmap slide. Run these five tasks against your own data before you sign anything. If a vendor is confident in what they’ve built, they’ll let you test it on real work orders and real specs, not just their sample dataset. If they push back on that request, that itself is useful information.
This article was written with the assistance of artificial intelligence. While we aim for accuracy, the information may be incomplete, out of date, or incorrect, and should be independently verified before you rely on it for any decision. It is provided for general information only and does not constitute professional advice.
